Alignment-Enriched Tuning for Patch-Level Pre-trained Document Image Models

Alignment between image and text has shown promising improvements on patch-level pre-trained document models. However, investigating more effective or finer-grained alignment techniques during pre-training requires a large amount of computation cost time. Thus, question naturally arises: Could we fine-tune the models adaptive to downstream tasks with objectives achieve comparable better performance? In this paper, propose new model architecture alignment-enriched tuning (dubbed AETNet) upon models, adapt joint task-specific supervised alignment-aware contrastive objective. Specifically, introduce an extra visual transformer as alignment-ware encoder before multimodal fusion. We consider in following three aspects: 1) document-level by leveraging cross-modal intra-modal loss; 2) global-local for modeling localized structural information images; 3) local-level accurate information. Experiments various show that AETNet can state-of-the-art performance tasks. Notably, consistently outperforms such LayoutLMv3 fine-tuning techniques, different Code is available at https://github.com/MAEHCM/AET.